Concurrent validity, cut‐offs and ability to change of patient‐reported outcome measures for rhinitis and asthma in MASK‐air®

Abstract Patient‐reported outcome measures (PROMs) are used to assess a patient's health status at a particular point in time. They are essential in the development of person‐centred care. This paper reviews studies performed on PROMs for assessing AR and asthma control, in particular VAS scales that are included in the app MASK‐air® (Mobile Airways Sentinel networK) for asthma and rhinitis. VASs were initially developed on paper and pencil and tested for their criterion validity, cut‐offs and responsiveness. Then, a multicentric, multinational, double‐blind, placebo‐controlled, randomised control trial (DB‐PC‐RCT) using an electronic VAS form was carried out. Finally, with the development of MASK‐air® in 2015, previously validated VAS questions were adapted to the digital format and further methodologic evaluations were performed. VAS for asthma, rhinitis, conjunctivitis, work and EQ‐5D are included in the app. Additionally, two control‐medication scores for allergic symptoms of asthma (e‐DASTHMA) were validated for their criterion validity, cut‐offs and responsiveness.

The first studies with VASs in AR and asthma were based on the paper-and-pencil format.6][7][8] In MASK-air ® , the same simple approach was used.We did not use individual symptoms of rhinitis and asthma because their pooling to propose a score should be validated using complex formulae to ascribe to each of the symptoms (e.g., rhinorrhoea, sneezing, nasal obstruction and/or pruritus) a weight in the burden and/or perception of the disease that may differ with age. 9,10Ss were then developed using paper and pencil and tested for their criterion validity, cut-offs and responsiveness.Then, a multicentric, multinational DB-PC-RCT using an electronic VAS form was carried out (Table 2).Finally, with the development of MASK-air ® (Mobile Airways Sentinel networK) in 2015, 24 previously validated VAS questions (Table 3) were adapted to the digital format and further methodologic evaluations were performed.
We also added to MASK-air ® the impact of the diseases on work 28 or education 29 as well as a generic VAS QOL questionnaire (EQ-5D-5L VAS). 25,30One question, "dyspnoea," was added temporarily to MASKair ® but the results showed a highly significant correlation with VAS 'asthma' in patients with severe asthma (rho = 0.93 and 0.90). 31,32The question was therefore subsequently deleted.

Strengths
� The VAS is more sensitive to small changes than simple descriptive ordinal scales on which symptoms are rated, for example, as mild, moderate or severe.� The VAS takes less than 1 minute to complete.� It is easy to use with routine treatment.� Training is not usually required to fill in a VAS, but some training may be needed when it is administered digitally.� It is acceptable for the patients.� It is inexpensive.
Weaknesses � The assessment is clearly highly subjective.� The VAS may be less valuable when comparing a group of patients at a one-time point.� The VAS is administered either as a paper-and-pencil measure or digitally.As a result, it cannot be administered verbally or by phone. 3� Test-retest reliability is good, but is higher among literate compared to illiterate patients 4 and among younger compared to older adults.� Drawbacks with paper-filled VASs do not exist when administered digitally.� In some diseases, in the absence of a gold standard, criterion validity cannot be evaluated.
We used either the total symptom score (TNSS, TOS or TSS), a validated QOL measure (RQLQ) or an asthma control score as gold standards to assess VAS construct validity in pre-MASK-air ® studies (Table 2).In MASK-air ® , we used the impact on work or academic productivity or utilities (EQ-5D-5L) as gold standards.In addition, we studied cut-off values and the ability to change before and during MASK-air ® studies.

| VAS in ARIA classes in an observational study in primary care (2004)
In study 1, Allergic Rhinitis and its Impact on Asthma (ARIA) 33 classes could be differentiated by VAS in AR patients seen in primary care. 11S scores ranged from not at all bothersome (0 cm) to extremely bothersome (10 cm).ARIA severity had more impact on VAS levels than duration.The receiver operating characteristic (ROC) curve identified a 5 cm cut-off for patients classified as 'mild' or 'moderate/ severe' rhinitis (negative predictive value: 93.5; positive predictive value: 73.6%).RQLQ global scores were moderately but significantly correlated (rho = 0.46; p < 0.0001).

| VAS in a multi-centre, cluster-randomised trial in specialists (2005)
In study 2, in patients with grass pollen-induced AR seen in specialist care in France, 12 a multicentre, cluster-randomised trial compared two therapeutic strategies.The RQLQ, the TNSS and VAS (0-10 cm) were self-reported by the patients before and after two weeks of treatment.A first post hoc analysis using ROC curves found that the optimal cut-off in VAS change separating patients without improvement from those with improvement was 0.30 cm. 13 By using cost function, a difference of more than 1 cm was significant.A second post VAS education a 'Today, how much did allergies affect your productivity while in school or attending classes in an academic setting?' EQ-5D-5L VAS EQ-5D-5L VAS 25 a Question derived from WPAI-AS 10,26,27 .
BOUSQUET ET AL.
hoc analysis compared TNSS, VAS and RQLQ before and after treatment. 12,14(Figure 1 and Table 4).VAS, RQLQ and TNSS showed similar change patterns and confirmed the ability to change VAS.

| VAS levels in a multi-centre observational study in primary care (2011)
The third study performed on patients consulting in primary care for AR assessed the impact of AR symptoms and the ARIA classes on QoL. 15 On day 14, treatments did not affect VAS levels.Changes in VAS levels were similar to those in TNSS4 and slightly lower than those in TSS6 or RQLQ.An a priori sub-analysis validated VAS as a simple quantitative tool to assess the burden of AR 16 and identified cut-off VAS levels for discriminating clinically relevant changes.The VAS cut-off variation of 23/100 mm was associated with a variation of 0.5 for RQLQ (minimal important difference in RQLQ).Sensitivity analysis with RQLQ and TSS6 scales confirmed the cut-off value.adolescents and adults consulting specialist physicians. 9,10In all age groups, the VAS score was strongly correlated with the weekly mean TSS score (Pearson's r: 0.79-0.88)and moderately correlated with the weekly mean RQLQ score (Pearson's r: 0.64-0.80).In moderateto-severe grass pollen-induced AR, symptom perception differed in children versus adolescents and adults.However, the assessments of treatment outcomes (using the TSS, VAS and RQLQ) were similar in all age groups.In an a priori sub-analysis including WPAI-AS, a multiple regression analysis indicated that the RQLQ score was a weak but statistically significant predictor of both impaired work/ classroom productivity and daily activities. 10
An a posteriori sub-analysis was carried out in 420 asthmatic patients using self-administered questionnaires, 20 VAS asthma and the Asthma Control Test (ACT) score.It showed a strong correlation (rho = −0.70,p < 0.001) (Figure 3).For an ACT<19 (cut-off for asthma control), most of the VAS asthma levels were over 20/100.
However, for an ACT>19, many patients had high VAS asthma levels.

| VAS 'GLOBAL ALLERGY SYMPTOMS' AND TOTAL SYMPTOM SCORE (TSS) USING AN ELECTRONIC DIARY IN 2 LARGE DB-PC-RCTS IN EUROPEAN AND CANADIAN SPECIALISTS (2008)
Two double-blind, placebo-controlled, randomised-controlled trials (DB-PC-RCTs) were conducted to assess whether desloratadine was effective in intermittent or persistent AR against placebo. 21,22,34,35I G U R E 2 Distribution of GINA control categories relative to VAS levels.

F I G U R E 3
Correlation between ACT and VAS asthma (SACRA) (from 20 ).
The first study assessed adolescents and adults with intermittent AR for 15 days 21 : desloratadine 5 mg once daily (n = 276) or placebo (n = 271).The primary endpoint, the AM/PM reflective five-symptom VAS, was compared with T5SS and displayed similar change patterns (Figure 4A).
The second study assessed adolescents and adults with perennial AR under treatment with desloratadine 5 mg once daily (n = 360) or placebo (n = 356) over an 85-day time period. 22The primary outcome was the AM/PM reflective TSS5 averaged over days 1-29.
Similar change patterns were observed for the TSS5 and VAS (Figure 4B).The first 2 weeks of analyses were similar in intermittent and perennial AR for both TSS and VAS, showing the strength of both approaches.

| VAS 'RHINO-CONJUNCTIVITIS SYMPTOMS', TSS, TNSS AND TOSS IN AN ALLERGEN-IMMUNOTHERAPY DP-PC-RCT IN EUROPE
A post-hoc analysis 23 was carried out from one pivotal DB-PC-RCT on immunotherapy with birch pollen. 36For each patient, TNSS, TOSS, TSS and VAS for rhinoconjunctivitis symptoms (VAS-RS)  5).Similar results were obtained when considering the full pollen season or the 7 worst days of the pollen season.

| MASK-AIR ® STUDIES (2015-)
MASK (Mobile Airway Sentinel networK), the Phase 3 ARIA 33,37 initiative, is a flexible e-platform for AR and asthma and includes the MASK-air ® app, which is operational in 30 countries and 20 languages (Table S1).Over 63,000 users have been registered.A description of MASK-air ® features is summarised in Figure 9.[50] Even though MASK-air ® is not the only multilingual app in rhinology (a market-research 51 identified three such apps: AllergyMonitor, Pollen Austria and MASK-air ® ), it is the best validated app for clinical practice and studies.

| Methodological approach
MASK-air ® has been validated using COSMIN guidelines and a series of methodologic studies. 52Limitations of MASK-air have been clearly defined. 52e Technology Readiness Levels (TRLs) 53 are usually eight or nine including PROMs for rhinitis and asthma, VAS, EQ-5D, CARAT, e-CSMS and e-DASTHMA.
T A B L E 5 Correlations between daily reported patient-reported outcome measures and RQLQ.

Correlations with the RQLQ (95% CI) a
Correlations with the VAS-RS (95% CI) a

F I G U R E 5
The MASK-air ® app.
BOUSQUET ET AL.

| Patient's acceptability
Two qualitative studies were carried out by MADOPA (Maintien en Autonomie à Domicile des Personnes Agées, https://www.madopa.fr) in 2016 to better understand the patients' needs and expectations. 54eir comments were embedded in MASK-air.
Five studies carried out in France (in preparation), Italy, 55 Lithuania, 56 Poland 57 and Portugal (submitted) showed that patients have an overall positive appreciation of MASK-air ® and proposed further improvements of the app.

| Impact of age on MASK-air ®
Studies assessing MASK-air ® data have traditionally included users ranging in age from 16 to over 90 years.Elderly users (≥65 years) can use the MASK-air ® app after a short training period. 55We assessed  The selection and validation of MASK-air ® VASs were based on studies using paper and pencil or e-diaries: VASs assessing for global allergy, nasal, ocular and asthma symptoms (Table 2). 31,59In addition, MASK-air ® assessed the daily impact of the allergies or asthma on work 28 or academic productivity 29 by means of VASs.A generic VAS on QOL (EQ-5D-5L VAS) was also included. 25,30reover, daily and personalised pollen counts and pollution data are available to complement VASs (Finish Meteorological Institute and Copernicus). 60 MASK-air ® , VAS asthma is highly correlated with VAS "dyspnoea" that was subsequently removed (Figure 6). 31,32all airways represent an important feature of asthma, and VAS asthma has been correlated with impulse oscillometry (IOS) in symptomatic patients. 61pilot trial evaluated the usefulness of the MASK-air ® app in improving rhinitis control in 262 patients with AR and asthma.62 There was a strong significant correlation between VAS asthma and ACT (Pearson coefficient: −0.79).In agreement, VAS asthma and ACT have shown correlation in a recent Turkish study (unpublished data).

| VAS cut-off values
In MASK-air, cut-offs were calculated using two different approaches in 395,223 days from 23,201 users: one based on the VAS percentiles and another based on VAS work and EQ-5D levels (Table 6).Strong correlations were found between VAS asthma and other measures (Table 7). 31,32I G U R E 6 Correlation between VAS asthma and dysponea in two studies on severe asthma (from 31,32 ).(allergy e-CSMS) 26 and asthma (e-DASTHMA). 48

Allergy e-CSMS
In 317,176 days of MASK-air ® use from 17,780 users aged 16-90 years in 25 countries, allergy e-CSMS was computed following cluster analysis, regression models or factor analysis (Figure 7).The selected CSMS displayed high accuracy (capacity of discriminating different levels of rhinitis control), and medium-high validity and reliability, rendering it as a candidate for primary endpoints in future rhinitis trials.Results were similar in different countries, showing the transferability and cultural adaptation of the CSMS.[67][68][69] e-DASTHMA An electronic daily asthma control score, e-DASTHMA, 48 has been developed in a cross-sectional study (35,635 days of MASK-air ® data, 1662 users).The developed score was strongly correlated with VAS dyspnoea and moderately correlated with other outcomes.It displayed high test-retest reliability and moderate-to-high responsiveness (Figure 8).
An external validation of e-DASTHMA was performed using an independent cohort of patients with physician-diagnosed asthma (INSPIRERS). 70e-DASTHMA was strongly correlated with the GINA classification of control.e-DASTHMA has been used in MASK-air ® studies. 71

| Other PROMs
Additional questionnaires are included in MASK-air ® : Control of Allergic Rhinitis and Asthma Test (CARAT, a PROM that assesses the level of control of both asthma and AR using a single tool over a period of 4 weeks), 72 WPAI-AS 73,74 and EQ-5D (Figure 7).However, other control tests can also be included in MASK-air ® .

T A B L E 7
Correlation coefficients between different PROMs in severe asthma (from 31 ).

| Work productivity
Rhinitis has a significant impact on work productivity. 75VAS work and WPAI-AS have been used in a series of concurrent validity studies (Table 8 and Figure 9).A significant correlation has been found between VAS work and WPAI-AS.reporting.There was a significant correlation between WPAI-AS and VAS education (rho = 0.80).Similar results were obtained when correlations were assessed using repeated measures correlation coefficients.

| EQ-5D-5L
Very few studies on allergic rhinitis and asthma have assessed each of the five EQ-5D domains individually.A cross-sectional MASK-air ® study assessed the association between rhinitis or asthma control and the different EQ-5D-5L domains in 5354 days from 3092 different users. 66Worse control of rhinitis, conjunctivitis or asthma (median VASs and CSMS) was associated with an impairment of EQ-5D levels.Mobility was particularly associated with VAS asthma (Figure 10).The expected differences in the EQ-5D 'mobility' domain between VAS asthma and the other comparators 79  -11 of 17 "VAS nose," "VAS eyes" and "VAS asthma") patients (Table 9). 78erall, results indicate that MASK-air ® VASs display a high testretest reliability, with ICC being always above 0.74.That is, VAS levels tend to display no or small changes in clinically stable patients.

| Ability to change
A pilot trial evaluated the usefulness of the MASK-air ® app in improving rhinitis control in 262 patients with AR. 62 VAS global changed similarly to TNSS.
In a quasi-experimental study in Greece, CARAT and MASK-air ® provided complementary information on AR symptom control, possibly mirroring differences in time periods assessed by these two tools (4 weeks vs. daily assessment).

| Transferability and cultural acceptability
MASK-air ® data are derived from up to 27 countries.When results were provided for each of the countries, there was a high correlation between them, including Brazil and Mexico.It can be concluded that MASK-air ® can be used in many countries and that the results are similar.However, for every new study, transferability would need to be confirmed since some questions may not be fully relevant in all countries.

3. 4 |T A B L E 4 Free
Abbreviations: ARIA, Allergic Rhinitis and its Impact on Asthma; RQLQ, Rhinoconjunctivitis Quality of Life Questionnaire; TNSS, Total Nasal Symptom Score; VAS, Visual analogue scale.a p < 0.001 by comparison to V1. b p < 0.001 by comparison to V1 and p < 0.01 by comparison to Free-treatment choice.
were administered by physicians.Self-and physicianadministered questionnaires produced similar results.Median VAS asthma levels were of 4.75 (25%-75%: 1.85-7.00) in the selfadministered questionnaire versus 4.79 (1.90-7.18) in the physician-administered questionnaire.Median VAS rhinitis levels were 5.49 (2.51-7.90) in the self-administered questionnaire versus 5.60 (2.75-7.85) in the physician-administered questionnaire.An a priori sub-analysis found that the VAS was a strong predictor of GINA-defined asthma. 19VAS asthma predicted levels of GINAdefined control categories (the area under the ROC curve ranged from 0.704 to 0.837), particularly with the VAS cut-offs of 1.50, 4.79 and 7.19.Similar results have been obtained using self-and physician-administered questionnaires showing the consistency of the results (Figure Evolution of T5SS and VAS global allergy symptoms in patients with intermittent allergic rhinitis treated with desloratadine or placebo.(B) Evolution of T5SS and VAS global allergy symptoms in patients with persistent allergic rhinitis treated with desloratadine or placebo.pollen season.Data from 482 subjects, providing 2937 daily records, were analysed.Daily VAS-RS was strongly correlated with the remaining daily PROMs and RQLQ score collected on the same day or in the respective weeks (Table

7. 4 . 3 |
Concurrent validity of VAS in MASK-air ®VAS asthma was studied using MASK-air ® data.Correlations between VAS asthma and other MASK-air ® daily reported PROMs were studied in severe asthmatic patients with nasal symptoms.

4 |
Combined electronic daily symptom- medication scores based on VAS Validated daily electronic combined symptom-medication scores (CSMSs) are needed to investigate the effects of AR or asthma treatments.MASK-air ® includes a medication list with all medications customised by countries.Combining medications and VAS allows the computation of electronic daily scores for allergic diseases

7. 6 F I G U R E 8
.2 | EducationAllergic diseases have a significant impact on academic performance and a MASK-air ® study compared VAS on allergy symptoms with VAS academic productivity.29In 13,454 days, there was a significant correlation between VAS education and VAS global (rho = 0.70) and VAS nose (rho = 0.66).The WPAI-AS provided data on the impact of allergies on school performance in 125 weeks with 6-7 days of Results between the best performing e-DASTHMA and comparators (from48

a
The entire population of asthmatic and non-asthmatic patients was compared.b In severe asthmatic patients only.F I G U R E 9 Correlation between VAS work and global allergy symptoms (from 77 N = 16,925 days).

8 | CONCLUSIONS 8 . 1 |
Pre-MASK-air ® studiesFor construct validity, in patients with rhinitis and/or asthma, the different VASs used have been shown to be highly correlated to RQLQ and total symptom scores for rhinitis and asthma control scores or to the GINA classification of control.Cut-off VAS values have been identified.The ability to detect a change was also found and correlated with TNSS or TSS and RQLQ.

8. 2 |
MASK-air ® studiesSimilar findings have been observed with MASK-air ® for daily PROMs.More data are needed for the ability to change.Importantly, similar results were found in different countries, indicating the transferability and cultural acceptability of MASK-air ® PROMs across countries.

8. 3 |
Clinical relevance of MASK-air ® PROMs and future studiesSimple PROMs that can be easily translated into several languages and do not need cultural adaptation are available in an electronic form in MASK-air ® .They are based on VASs that were found to be of clinical relevance.The daily electronic combined symptom-medication scores represent an improvement over VAS as they represent both PROMs and medications.However, these tools need to be tested to determine their relevance in the stratification of severe patients and the costeffective management of rhinitis and/or asthma to provide recommendations for policy makers.Preliminary work has been developed to assess how MASK-air ® PROMs can be used for patient stratification.In fact, we assessed complete months in which patients under inhaled corticosteroids had reported VAS asthma and e-DASTHMA levels, identifying groups of patients who would most probably benefit from treatment scaling up versus those who would benefit from increased adherence (Sousa-Pinto et al., in preparation).This can support efforts to increase the efficiency in the management of allergic diseases.Finally, these PROMs are currently being tested for their value in the diagnosis of under-recognised patients.
Visual analogue scales used for the daily monitoring of symptoms and impact in all studies and MASK-air ® .
T A B L E 2 Studies performed to validate the MASK-air ® VASs.
(Figures 5 and S1).MASK-air ® , a validated mHealth app (Medical Device regulation Class IIa), is a Good Practice of DG Santé on digitally enabled personcentred care 39 and a Best Practice of OECD (Organisation for Economic Co-operation and Development) for Public Health guidelines in chronic diseases.The vision of MASK-air ® has led to a strategic overview that was initiated by ARIA in 1999.It includes WHOassociated projects, 63SK-air cut-off levels (/100) (from63).

| Intra-rater reliability
78rengthen the validity of VAS asthma.On the other hand, VAS asthma is less well correlated with the EQ-5D 'daily activities' domain, in accordance with the results of the European Community Respiratory Health Between 2412 ("VAS work") and 5827 ("VAS nose" and "VAS eyes") days with more than one daily monitoring VAS input provided by the same user were recorded.78ForallVASs,more than 50% of the days had no differences in the first and second values provided within the same day.Differences between the first and second daily values Test-retest analyses from 25 countries (from78).
7.8 | Test-retest reliabilityUsing clinical stability defined according to several outcomes, the test-retest reliability of daily monitoring VASs was assessed based on data from 102 ("VAS work") to 270 ("VAS global allergy symptoms," T A B L E 9 F I G U R E 1 0 Association of VAS or CSMS with EQ-5D-5L domains in a multivariate analysis (from 66 ).BOUSQUET ET AL.